span-crossref-sync

Overview ¶

span-crossref-sync download caches raw crossref messages from the crossref works API.

Example usage:

$ span-crossref-sync -p zstd \ # compress program
  -P feed-1- \                 # file prefix
  -i d \                       # interval (daily)
  -verbose \                   # verbose
  -t 30m \                     # timeout
  -s 2022-01-01 \              # start
  -e 2023-05-01 \              # end
  -c /data/finc/crossref/      # cache dir

Space requirements: One day yields about 1M update docs, or a ~2GB compressed file. A year equates to about 800G of compressed data.

This can run independently of other conversion processes, e.g. in a daily cron job. Processes that need this data can manually find files or create a snapshot.

Data point: https://github.com/miku/filterline#data-point-crossref-snapshot