Documentation
¶
Overview ¶
Package crossrefcmd holds the library code backing the span-crossref-* commands: table, fast-snapshot, fastproc, members, snapshot and sync.
Index ¶
- func BuildTagger(opts FilterConfigOpts) (tagger *filter.Tagger, cleanup func(), err error)
- func CopyFile(dst, src string, perm os.FileMode) (err error)
- func OutputFilename(inputPath string) string
- func ParseExcludes(r io.Reader) ([]string, error)
- func ProcessFile(cfg FastprocConfig, tagger *filter.Tagger, inputPath string, w io.Writer) error
- func ProcessStream(cfg FastprocConfig, tagger *filter.Tagger, r io.Reader, w io.Writer) error
- func ResolveFilterConfig(opts FilterConfigOpts) (string, error)
- func RunFastSnapshot(cfg FastSnapshotConfig, snapshot SnapshotFunc) error
- func RunMembers(cfg MembersConfig, get HTTPGetter, w io.Writer) error
- func RunTable(cfg TableConfig, r io.Reader, w io.Writer) error
- func Stage1Extract(cfg Stage1Config, excludes map[string]struct{}, r io.Reader, w io.Writer) error
- func WriteFields(w io.Writer, sep string, values ...any) (int, error)
- type Doer
- type FastSnapshotConfig
- type FastprocConfig
- type FilterConfigOpts
- type HTTPGetter
- type MembersConfig
- type MembersResponse
- type SnapshotFunc
- type Stage1Config
- type Sync
- type TableConfig
- type WorksResponse
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func BuildTagger ¶
func BuildTagger(opts FilterConfigOpts) (tagger *filter.Tagger, cleanup func(), err error)
BuildTagger resolves the filterconfig (frozen file or FOLIO), loads it into a Tagger and applies any meta-ISIL expansion rules. Expansion is always applied to the in-memory tagger, so the frozen-file and FOLIO paths behave identically and the FOLIO cache stays independent of the expand rules. The returned cleanup removes the temporary directory the config was unfrozen into; callers should defer it.
func CopyFile ¶
CopyFile copies the contents from src to dst using io.Copy. If dst does not exist, CopyFile creates it with permissions perm; otherwise CopyFile truncates it before writing. From: https://codereview.appspot.com/152180043
func OutputFilename ¶
OutputFilename derives the output filename from the input filename. Only .json.zst input is supported. feed-2-index-2026-03-02-2026-03-02.json.zst -> feed-2-index-2026-03-02-2026-03-02-solr-export-with-fullrecord.json.zst
func ParseExcludes ¶
ParseExcludes reads DOIs to exclude, one per line, from r. As in the original, the result is the newline-split content (a trailing empty element is possible).
func ProcessFile ¶
ProcessFile opens a zstd-compressed crossref slice at inputPath and streams the resulting solr records to w (see ProcessStream).
func ProcessStream ¶
ProcessStream runs the equivalent of "span-import -i crossref | span-tag -unfreeze filterconfig.zip | span-export -with-fullrecord" over the records read from r (decompressed crossref, one JSON per line), writing solr records to w.
func ResolveFilterConfig ¶
func ResolveFilterConfig(opts FilterConfigOpts) (string, error)
ResolveFilterConfig returns the path to a frozen filterconfig zip. If opts.FrozenFile is set, it returns that path. Otherwise it fetches from FOLIO with caching.
Expansion (opts.ExpandFlag) is intentionally not applied here: it is applied to the in-memory tagger in BuildTagger. Baking it into the fetched config would make the cached zip depend on the expand rules, which the FOLIO cache key does not capture.
func RunFastSnapshot ¶
func RunFastSnapshot(cfg FastSnapshotConfig, snapshot SnapshotFunc) error
RunFastSnapshot creates the snapshot using snapshot (defaults to crossref.CreateSnapshot when nil).
func RunMembers ¶
func RunMembers(cfg MembersConfig, get HTTPGetter, w io.Writer) error
RunMembers paginates through the crossref members API and writes one raw response per line to w. On HTTP >= 400 it retries (up to cfg.RetryCount) with a fixed backoff.
func RunTable ¶
RunTable reads crossref documents (one JSON per line) from r and writes the tabular representation to w.
func Stage1Extract ¶
Stage1Extract reads raw crossref works messages (one JSON per line) from r and writes a tab-separated "lineno<TAB>indexed-date<TAB>DOI" record for each non-excluded document to w. Up to cfg.ErrCountThreshold JSON/date errors are tolerated (logged and skipped); beyond that the error is returned.
Types ¶
type Doer ¶
Doer abstracts https://pkg.go.dev/net/http#Client.Do.
type FastSnapshotConfig ¶
type FastSnapshotConfig struct {
InputFiles []string
OutputFile string
BatchSize int
NumWorkers int
Verbose bool
KeepTempFiles bool
SortBufferSize string
Excludes []string
ShuffleInputFiles bool
CacheEnabled bool
CacheDir string
CacheClear bool
}
FastSnapshotConfig mirrors the span-crossref-fast-snapshot flags. InputFiles and Excludes are the resolved values (main reads the excludes file into Excludes).
func DefaultFastSnapshotConfig ¶
func DefaultFastSnapshotConfig() FastSnapshotConfig
DefaultFastSnapshotConfig returns the default configuration for RunFastSnapshot.
func (FastSnapshotConfig) Options ¶
func (c FastSnapshotConfig) Options() crossref.SnapshotOptions
Options maps the config to crossref.SnapshotOptions.
type FastprocConfig ¶
FastprocConfig holds the tunables for RunFastproc / ProcessStream.
func DefaultFastprocConfig ¶
func DefaultFastprocConfig() FastprocConfig
DefaultFastprocConfig returns the default configuration.
type FilterConfigOpts ¶
type FilterConfigOpts struct {
FrozenFile string
OkapiURL string
Tenant string
Token string
ExpandFlag string
NoProxy bool
CacheTTL time.Duration
Force bool
}
FilterConfigOpts describes where to obtain the filterconfig zip. If FrozenFile is set it is returned directly; otherwise the config is fetched from FOLIO (with caching).
type HTTPGetter ¶
HTTPGetter fetches a URL. It is injected so RunMembers can be tested without hitting the network; the default is http.Get.
type MembersConfig ¶
MembersConfig holds the pagination options for RunMembers.
func DefaultMembersConfig ¶
func DefaultMembersConfig() MembersConfig
DefaultMembersConfig returns the default configuration.
type MembersResponse ¶
type MembersResponse struct {
Message struct {
Items []struct {
Breakdowns any `json:"breakdowns"`
Counts struct {
BackfileDois int64 `json:"backfile-dois"`
CurrentDois int64 `json:"current-dois"`
TotalDois int64 `json:"total-dois"`
} `json:"counts"`
CountsType any `json:"counts-type"`
Coverage any `json:"coverage"`
CoverageType any `json:"coverage-type"`
Flags struct {
Deposits bool `json:"deposits"`
DepositsAbstractsBackfile bool `json:"deposits-abstracts-backfile"`
DepositsAbstractsCurrent bool `json:"deposits-abstracts-current"`
DepositsAffiliationsBackfile bool `json:"deposits-affiliations-backfile"`
DepositsAffiliationsCurrent bool `json:"deposits-affiliations-current"`
DepositsArticles bool `json:"deposits-articles"`
DepositsAwardNumbersBackfile bool `json:"deposits-award-numbers-backfile"`
DepositsAwardNumbersCurrent bool `json:"deposits-award-numbers-current"`
DepositsFundersBackfile bool `json:"deposits-funders-backfile"`
DepositsFundersCurrent bool `json:"deposits-funders-current"`
DepositsLicensesBackfile bool `json:"deposits-licenses-backfile"`
DepositsLicensesCurrent bool `json:"deposits-licenses-current"`
DepositsOpenReferencesBackfile bool `json:"deposits-open-references-backfile"`
DepositsOpenReferencesCurrent bool `json:"deposits-open-references-current"`
DepositsOrcidsBackfile bool `json:"deposits-orcids-backfile"`
DepositsOrcidsCurrent bool `json:"deposits-orcids-current"`
DepositsReferencesBackfile bool `json:"deposits-references-backfile"`
DepositsReferencesCurrent bool `json:"deposits-references-current"`
DepositsResourceLinksBackfile bool `json:"deposits-resource-links-backfile"`
DepositsResourceLinksCurrent bool `json:"deposits-resource-links-current"`
DepositsSimilarityCheckingBackfile bool `json:"deposits-similarity-checking-backfile"`
DepositsSimilarityCheckingCurrent bool `json:"deposits-similarity-checking-current"`
DepositsUpdatePoliciesBackfile bool `json:"deposits-update-policies-backfile"`
DepositsUpdatePoliciesCurrent bool `json:"deposits-update-policies-current"`
} `json:"flags"`
Id int64 `json:"id"`
LastStatusCheckTime int64 `json:"last-status-check-time"`
Location string `json:"location"`
Names []string `json:"names"`
Prefix []struct {
Name string `json:"name"`
PublicReferences bool `json:"public-references"`
ReferenceVisibility string `json:"reference-visibility"`
Value string `json:"value"`
} `json:"prefix"`
Prefixes []string `json:"prefixes"`
PrimaryName string `json:"primary-name"`
Tokens []string `json:"tokens"`
} `json:"items"`
ItemsPerPage int64 `json:"items-per-page"`
Query struct {
SearchTerms any `json:"search-terms"`
StartIndex int64 `json:"start-index"`
} `json:"query"`
TotalResults int64 `json:"total-results"`
} `json:"message"`
MessageType string `json:"message-type"`
MessageVersion string `json:"message-version"`
Status string `json:"status"`
}
MembersResponse from api.crossref.org/member, generated by JSONGen, use offset to paginate.
type SnapshotFunc ¶
type SnapshotFunc func(crossref.SnapshotOptions) error
SnapshotFunc creates a snapshot from options. It is injected so option mapping can be tested without running the (heavy) real snapshot.
type Stage1Config ¶
Stage1Config holds the tunables for Stage1Extract.
func DefaultStage1Config ¶
func DefaultStage1Config() Stage1Config
DefaultStage1Config returns the default configuration.
type Sync ¶
type Sync struct {
ApiEndpoint string
ApiFilter string
ApiEmail string
Rows int
UserAgent string
Client Doer
Verbose bool
Mode string
MaxRetries int
}
Sync saves messages from crossref.
func (*Sync) WriteWindow ¶
WriteWindow writes a slice of data from the API to a writer. The dates in filters should always be of the form YYYY-MM-DD, YYYY-MM or YYYY. The date filters are inclusive (https://api.crossref.org/swagger-ui/index.html#/operations/Works/get_works).
type TableConfig ¶
TableConfig holds the tunables for a tabularization run.
func DefaultTableConfig ¶
func DefaultTableConfig() TableConfig
DefaultTableConfig returns the default configuration for RunTable.
type WorksResponse ¶
type WorksResponse struct {
Message struct {
Facets struct {
} `json:"facets"`
Items []json.RawMessage `json:"items"`
ItemsPerPage int64 `json:"items-per-page"`
NextCursor string `json:"next-cursor"` // iterate
Query struct {
SearchTerms any `json:"search-terms"`
StartIndex int64 `json:"start-index"`
} `json:"query"`
TotalResults int64 `json:"total-results"` // want to estimate total results (and verify download)
} `json:"message"`
MessageType string `json:"message-type"`
MessageVersion string `json:"message-version"`
Status string `json:"status"`
}
WorksResponse, stripped of the actual messages, as we only need the status and mayby total results.