crossrefcmd

package
v0.2.37 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 20, 2026 License: GPL-3.0 Imports: 24 Imported by: 0

Documentation

Overview

Package crossrefcmd holds the library code backing the span-crossref-* commands: table, fast-snapshot, fastproc, members, snapshot and sync.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func BuildTagger

func BuildTagger(opts FilterConfigOpts) (tagger *filter.Tagger, cleanup func(), err error)

BuildTagger resolves the filterconfig (frozen file or FOLIO), loads it into a Tagger and applies any meta-ISIL expansion rules. Expansion is always applied to the in-memory tagger, so the frozen-file and FOLIO paths behave identically and the FOLIO cache stays independent of the expand rules. The returned cleanup removes the temporary directory the config was unfrozen into; callers should defer it.

func CopyFile

func CopyFile(dst, src string, perm os.FileMode) (err error)

CopyFile copies the contents from src to dst using io.Copy. If dst does not exist, CopyFile creates it with permissions perm; otherwise CopyFile truncates it before writing. From: https://codereview.appspot.com/152180043

func OutputFilename

func OutputFilename(inputPath string) string

OutputFilename derives the output filename from the input filename. Only .json.zst input is supported. feed-2-index-2026-03-02-2026-03-02.json.zst -> feed-2-index-2026-03-02-2026-03-02-solr-export-with-fullrecord.json.zst

func ParseExcludes

func ParseExcludes(r io.Reader) ([]string, error)

ParseExcludes reads DOIs to exclude, one per line, from r. As in the original, the result is the newline-split content (a trailing empty element is possible).

func ProcessFile

func ProcessFile(cfg FastprocConfig, tagger *filter.Tagger, inputPath string, w io.Writer) error

ProcessFile opens a zstd-compressed crossref slice at inputPath and streams the resulting solr records to w (see ProcessStream).

func ProcessStream

func ProcessStream(cfg FastprocConfig, tagger *filter.Tagger, r io.Reader, w io.Writer) error

ProcessStream runs the equivalent of "span-import -i crossref | span-tag -unfreeze filterconfig.zip | span-export -with-fullrecord" over the records read from r (decompressed crossref, one JSON per line), writing solr records to w.

func ResolveFilterConfig

func ResolveFilterConfig(opts FilterConfigOpts) (string, error)

ResolveFilterConfig returns the path to a frozen filterconfig zip. If opts.FrozenFile is set, it returns that path. Otherwise it fetches from FOLIO with caching.

Expansion (opts.ExpandFlag) is intentionally not applied here: it is applied to the in-memory tagger in BuildTagger. Baking it into the fetched config would make the cached zip depend on the expand rules, which the FOLIO cache key does not capture.

func RunFastSnapshot

func RunFastSnapshot(cfg FastSnapshotConfig, snapshot SnapshotFunc) error

RunFastSnapshot creates the snapshot using snapshot (defaults to crossref.CreateSnapshot when nil).

func RunMembers

func RunMembers(cfg MembersConfig, get HTTPGetter, w io.Writer) error

RunMembers paginates through the crossref members API and writes one raw response per line to w. On HTTP >= 400 it retries (up to cfg.RetryCount) with a fixed backoff.

func RunTable

func RunTable(cfg TableConfig, r io.Reader, w io.Writer) error

RunTable reads crossref documents (one JSON per line) from r and writes the tabular representation to w.

func Stage1Extract

func Stage1Extract(cfg Stage1Config, excludes map[string]struct{}, r io.Reader, w io.Writer) error

Stage1Extract reads raw crossref works messages (one JSON per line) from r and writes a tab-separated "lineno<TAB>indexed-date<TAB>DOI" record for each non-excluded document to w. Up to cfg.ErrCountThreshold JSON/date errors are tolerated (logged and skipped); beyond that the error is returned.

func WriteFields

func WriteFields(w io.Writer, sep string, values ...any) (int, error)

WriteFields writes a variable number of values separated by sep to a given writer. Returns bytes written and error.

Types

type Doer

type Doer interface {
	Do(*http.Request) (*http.Response, error)
}

Doer abstracts https://pkg.go.dev/net/http#Client.Do.

type FastSnapshotConfig

type FastSnapshotConfig struct {
	InputFiles        []string
	OutputFile        string
	BatchSize         int
	NumWorkers        int
	Verbose           bool
	KeepTempFiles     bool
	SortBufferSize    string
	Excludes          []string
	ShuffleInputFiles bool
	CacheEnabled      bool
	CacheDir          string
	CacheClear        bool
}

FastSnapshotConfig mirrors the span-crossref-fast-snapshot flags. InputFiles and Excludes are the resolved values (main reads the excludes file into Excludes).

func DefaultFastSnapshotConfig

func DefaultFastSnapshotConfig() FastSnapshotConfig

DefaultFastSnapshotConfig returns the default configuration for RunFastSnapshot.

func (FastSnapshotConfig) Options

Options maps the config to crossref.SnapshotOptions.

type FastprocConfig

type FastprocConfig struct {
	NumWorkers int
	BatchSize  int
}

FastprocConfig holds the tunables for RunFastproc / ProcessStream.

func DefaultFastprocConfig

func DefaultFastprocConfig() FastprocConfig

DefaultFastprocConfig returns the default configuration.

type FilterConfigOpts

type FilterConfigOpts struct {
	FrozenFile string
	OkapiURL   string
	Tenant     string
	Token      string
	ExpandFlag string
	NoProxy    bool
	CacheTTL   time.Duration
	Force      bool
}

FilterConfigOpts describes where to obtain the filterconfig zip. If FrozenFile is set it is returned directly; otherwise the config is fetched from FOLIO (with caching).

type HTTPGetter

type HTTPGetter func(url string) (*http.Response, error)

HTTPGetter fetches a URL. It is injected so RunMembers can be tested without hitting the network; the default is http.Get.

type MembersConfig

type MembersConfig struct {
	Offset     int
	Rows       int
	Base       string
	Sleep      time.Duration
	RetryCount int
}

MembersConfig holds the pagination options for RunMembers.

func DefaultMembersConfig

func DefaultMembersConfig() MembersConfig

DefaultMembersConfig returns the default configuration.

type MembersResponse

type MembersResponse struct {
	Message struct {
		Items []struct {
			Breakdowns any `json:"breakdowns"`
			Counts     struct {
				BackfileDois int64 `json:"backfile-dois"`
				CurrentDois  int64 `json:"current-dois"`
				TotalDois    int64 `json:"total-dois"`
			} `json:"counts"`
			CountsType   any `json:"counts-type"`
			Coverage     any `json:"coverage"`
			CoverageType any `json:"coverage-type"`
			Flags        struct {
				Deposits                           bool `json:"deposits"`
				DepositsAbstractsBackfile          bool `json:"deposits-abstracts-backfile"`
				DepositsAbstractsCurrent           bool `json:"deposits-abstracts-current"`
				DepositsAffiliationsBackfile       bool `json:"deposits-affiliations-backfile"`
				DepositsAffiliationsCurrent        bool `json:"deposits-affiliations-current"`
				DepositsArticles                   bool `json:"deposits-articles"`
				DepositsAwardNumbersBackfile       bool `json:"deposits-award-numbers-backfile"`
				DepositsAwardNumbersCurrent        bool `json:"deposits-award-numbers-current"`
				DepositsFundersBackfile            bool `json:"deposits-funders-backfile"`
				DepositsFundersCurrent             bool `json:"deposits-funders-current"`
				DepositsLicensesBackfile           bool `json:"deposits-licenses-backfile"`
				DepositsLicensesCurrent            bool `json:"deposits-licenses-current"`
				DepositsOpenReferencesBackfile     bool `json:"deposits-open-references-backfile"`
				DepositsOpenReferencesCurrent      bool `json:"deposits-open-references-current"`
				DepositsOrcidsBackfile             bool `json:"deposits-orcids-backfile"`
				DepositsOrcidsCurrent              bool `json:"deposits-orcids-current"`
				DepositsReferencesBackfile         bool `json:"deposits-references-backfile"`
				DepositsReferencesCurrent          bool `json:"deposits-references-current"`
				DepositsResourceLinksBackfile      bool `json:"deposits-resource-links-backfile"`
				DepositsResourceLinksCurrent       bool `json:"deposits-resource-links-current"`
				DepositsSimilarityCheckingBackfile bool `json:"deposits-similarity-checking-backfile"`
				DepositsSimilarityCheckingCurrent  bool `json:"deposits-similarity-checking-current"`
				DepositsUpdatePoliciesBackfile     bool `json:"deposits-update-policies-backfile"`
				DepositsUpdatePoliciesCurrent      bool `json:"deposits-update-policies-current"`
			} `json:"flags"`
			Id                  int64    `json:"id"`
			LastStatusCheckTime int64    `json:"last-status-check-time"`
			Location            string   `json:"location"`
			Names               []string `json:"names"`
			Prefix              []struct {
				Name                string `json:"name"`
				PublicReferences    bool   `json:"public-references"`
				ReferenceVisibility string `json:"reference-visibility"`
				Value               string `json:"value"`
			} `json:"prefix"`
			Prefixes    []string `json:"prefixes"`
			PrimaryName string   `json:"primary-name"`
			Tokens      []string `json:"tokens"`
		} `json:"items"`
		ItemsPerPage int64 `json:"items-per-page"`
		Query        struct {
			SearchTerms any   `json:"search-terms"`
			StartIndex  int64 `json:"start-index"`
		} `json:"query"`
		TotalResults int64 `json:"total-results"`
	} `json:"message"`
	MessageType    string `json:"message-type"`
	MessageVersion string `json:"message-version"`
	Status         string `json:"status"`
}

MembersResponse from api.crossref.org/member, generated by JSONGen, use offset to paginate.

type SnapshotFunc

type SnapshotFunc func(crossref.SnapshotOptions) error

SnapshotFunc creates a snapshot from options. It is injected so option mapping can be tested without running the (heavy) real snapshot.

type Stage1Config

type Stage1Config struct {
	BatchSize         int
	ErrCountThreshold int64
}

Stage1Config holds the tunables for Stage1Extract.

func DefaultStage1Config

func DefaultStage1Config() Stage1Config

DefaultStage1Config returns the default configuration.

type Sync

type Sync struct {
	ApiEndpoint string
	ApiFilter   string
	ApiEmail    string
	Rows        int
	UserAgent   string
	Client      Doer
	Verbose     bool
	Mode        string
	MaxRetries  int
}

Sync saves messages from crossref.

func (*Sync) WriteWindow

func (s *Sync) WriteWindow(w io.Writer, f, u time.Time) error

WriteWindow writes a slice of data from the API to a writer. The dates in filters should always be of the form YYYY-MM-DD, YYYY-MM or YYYY. The date filters are inclusive (https://api.crossref.org/swagger-ui/index.html#/operations/Works/get_works).

type TableConfig

type TableConfig struct {
	BatchSize  int
	NumWorkers int
}

TableConfig holds the tunables for a tabularization run.

func DefaultTableConfig

func DefaultTableConfig() TableConfig

DefaultTableConfig returns the default configuration for RunTable.

type WorksResponse

type WorksResponse struct {
	Message struct {
		Facets struct {
		} `json:"facets"`
		Items        []json.RawMessage `json:"items"`
		ItemsPerPage int64             `json:"items-per-page"`
		NextCursor   string            `json:"next-cursor"` // iterate
		Query        struct {
			SearchTerms any   `json:"search-terms"`
			StartIndex  int64 `json:"start-index"`
		} `json:"query"`
		TotalResults int64 `json:"total-results"` // want to estimate total results (and verify download)
	} `json:"message"`
	MessageType    string `json:"message-type"`
	MessageVersion string `json:"message-version"`
	Status         string `json:"status"`
}

WorksResponse, stripped of the actual messages, as we only need the status and mayby total results.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL